Neural correlates of category learning in monkey inferior temporal cortex.

We trained two monkeys implanted with multi-electrode arrays to categorize natural images of cats and dogs, in order to observe changes in neural activity related to category learning. We recorded neural activity from area TE, which is required for normal learning of visual categories based on perceptual similarity. Neural activity during a passive viewing task was compared pre- and post-training. After the category training, the accuracy of abstract category decoding improved. Specifically, the proportion of single units with category selectivity increased, and units sustained their category-specific responses for longer. Visual category learning thus appears to enhance category separability in area TE by driving changes in the stimulus selectivity of individual neurons and by recruiting more units to the active network.

Recurrent Connections Might Be Important for Hierarchical Categorization.

Visual short-term memory is an important ability of primates and is thought to be stored in area TE. We previously reported that the initial transient responses of neurons in area TE represented information about a global category of faces, e.g., monkey faces vs. human faces vs. simple shapes, and the latter part of the responses represented information about fine categories, e.g., facial expression. The neuronal mechanisms of hierarchical categorization in area TE remain unknown. For this study, we constructed a combined model that consisted of a deep neural network (DNN) and a recurrent neural network and investigated whether this model can replicate the time course of hierarchical categorization. The visual images were stored in the recurrent connections of the model. When the visual images with noise were input to the model, the model outputted the time course of the hierarchical categorization. This result indicates that recurrent connections in the model are important not only for visual short-term memory but for hierarchical categorization, suggesting that recurrent connections in area TE are important for hierarchical categorization.

A convolutional neural network for estimating synaptic connectivity from spike trains.

The recent increase in reliable, simultaneous high channel count extracellular recordings is exciting for physiologists and theoreticians because it offers the possibility of reconstructing the underlying neuronal circuits. We recently presented a method of inferring this circuit connectivity from neuronal spike trains by applying the generalized linear model to cross-correlograms. Although the algorithm can do a good job of circuit reconstruction, the parameters need to be carefully tuned for each individual dataset. Here we present another method using a Convolutional Neural Network for Estimating synaptic Connectivity from spike trains. After adaptation to huge amounts of simulated data, this method robustly captures the specific feature of monosynaptic impact in a noisy cross-correlogram. There are no user-adjustable parameters. With this new method, we have constructed diagrams of neuronal circuits recorded in several cortical areas of monkeys.

Comparison of neuronal responses in primate inferior-temporal cortex and feed-forward deep neural network model with regard to information processing of faces.

Feed-forward deep neural networks have better performance in object categorization tasks than other models of computer vision. To understand the relationship between feed-forward deep networks and the primate brain, we investigated representations of upright and inverted faces in a convolutional deep neural network model and compared them with representations by neurons in the monkey anterior inferior-temporal cortex, area TE. We applied principal component analysis to feature vectors in each model layer to visualize the relationship between the vectors of the upright and inverted faces. The vectors of the upright and inverted monkey faces were more separated through the convolution layers. In the fully-connected layers, the separation among human individuals for upright faces was larger than for inverted faces. The Spearman correlation between each model layer and TE neurons reached a maximum at the fully-connected layers. These results indicate that the processing of faces in the fully-connected layers might resemble the asymmetric representation of upright and inverted faces by the TE neurons. The separation of upright and inverted faces might take place by feed-forward processing in the visual cortex, and separations among human individuals for upright faces, which were larger than those for inverted faces, might occur in area TE.

Recency memory effects in Macaques during sequential delayed match-to-sample task with visual noise.

Visual object recognition requires both visual sensory information and memory, and its mechanisms are often studied using old-world monkeys. Wittig et al. (2014, 2016) reported that Rhesus monkeys and humans seem to adopt different strategies in a short-term visual memory task. The Rhesus monkeys seemed to rely on recency of stimulus repetition, whereas humans relied on specific memorization. We conducted experiments using a sequential delayed match-to-sample task with random dot visual noise using Rhesus and Japanese monkeys and found that recency effect was observed in both species. There were differences in the noise effect on behavioral performances across species.

Information Accumulation over Time in Monkey Inferior Temporal Cortex Neurons Explains Pattern Recognition Reaction Time under Visual Noise.

We recognize objects even when they are partially degraded by visual noise. We studied the relation between the amount of visual noise (5, 10, 15, 20, or 25%) degrading 8 black-and-white stimuli and stimulus identification in 2 monkeys performing a sequential delayed match-to-sample task. We measured the accuracy and speed with which matching stimuli were identified. The performance decreased slightly (errors increased) as the amount of visual noise increased for both monkeys. The performance remained above 80% correct, even with 25% noise. However, the reaction times markedly increased as the noise increased, indicating that the monkeys took progressively longer to decide what the correct response would be as the amount of visual noise increased, showing that the monkeys trade time to maintain accuracy. Thus, as time unfolds the monkeys act as if they are accumulating the information and/or testing hypotheses about whether the test stimulus is likely to be a match for the sample being held in short-term memory. We recorded responses from 13 single neurons in area TE of the 2 monkeys. We found that stimulus-selective information in the neuronal responses began accumulating when the match stimulus appeared. We found that the greater the amount of noise obscuring the test stimulus, the more slowly stimulus-related information by the 13 neurons accumulated. The noise induced slowing was about the same for both behavior and information. These data are consistent with the hypothesis that area TE neuron population carries information about stimulus identity that accumulates over time in such a manner that it progressively overcomes the signal degradation imposed by adding visual noise.

[Neural Mechanisms Underlying the Face Inversion Effect].

The ability to recognize faces is reduced with a picture-plane inversion of the faces, known as the face inversion effect. It has been reported that the configuration of facial features, for example, the distance between the eyes and mouth, becomes less perceptible when the face is inverted. In macaque monkeys, designated cortical areas, i.e., face patches, where face images are processed, have been found in the temporal visual cortex along the ventral visual pathway. Neurons in the anterior face patch (anterior part of the inferior temporal cortex) are known to encode view-invariant identity information. Thus, the anterior face patch is believed to be the final processing stage in the face patch system. A recent study showed that the face-inversion decreases the amount of the information about facial identity and facial expression conveyed by neurons, though it did not affect the information about the global category of the stimulus images (monkey versus human versus shape). The anterior face patch may, therefore, serve as the neural basis underlying the face inversion effect.

Face inversion decreased information about facial identity and expression in face-responsive neurons in macaque area TE.

To investigate the effect of face inversion and thatcherization (eye inversion) on temporal processing stages of facial information, single neuron activities in the temporal cortex (area TE) of two rhesus monkeys were recorded. Test stimuli were colored pictures of monkey faces (four with four different expressions), human faces (three with four different expressions), and geometric shapes. Modifications were made in each face-picture, and its four variations were used as stimuli: upright original, inverted original, upright thatcherized, and inverted thatcherized faces. A total of 119 neurons responded to at least one of the upright original facial stimuli. A majority of the neurons (71%) showed activity modulations depending on upright and inverted presentations, and a lesser number of neurons (13%) showed activity modulations depending on original and thatcherized face conditions. In the case of face inversion, information about the fine category (facial identity and expression) decreased, whereas information about the global category (monkey vs human vs shape) was retained for both the original and thatcherized faces. Principal component analysis on the neuronal population responses revealed that the global categorization occurred regardless of the face inversion and that the inverted faces were represented near the upright faces in the principal component analysis space. By contrast, the face inversion decreased the ability to represent human facial identity and monkey facial expression. Thus, the neuronal population represented inverted faces as faces but failed to represent the identity and expression of the inverted faces, indicating that the neuronal representation in area TE cause the perceptual effect of face inversion.

Small effect of upcoming reward outcomes on visual cue-related neuronal activity in macaque area TE during conditional associations.

Area TE sends dense projections to the perirhinal cortex in macaque monkeys, an area in which we previously observed flexible signals regarding upcoming reward outcomes during a conditional-association cued-reward task. To investigate neuronal processing during the generation of information on upcoming reward outcomes, neuronal activities in area TE were examined. In the task, a color stimulus as Cue 1 and a pattern stimulus as Cue 2 were sequentially presented. Each pattern stimulus indicated both rewarded and unrewarded outcomes depending on the preceding color stimulus. In the activities during Cue 2 presentation, two-way analysis of variance revealed the effect of the interaction between Cue 1 and Cue 2, i.e., reward conditions, in 19 out of 133 neurons recorded in area TE. Of the 19 neurons, 12 also represented a response delineating a specific cue sequence, i.e., a trial-type activity. The latency of the reward-condition dependence in 7 neurons without the trial-type activity was indistinguishable from the latency in neurons without a trial-type activity in the perirhinal cortex. These results suggest that the effect of upcoming reward conditions is small in area TE and that the representation of reward conditions arises in areas beyond the ventral visual pathway, including the perirhinal cortex, during conditional associations.

Stimulus-related activity during conditional associations in monkey perirhinal cortex neurons depends on upcoming reward outcome.

Acquiring the significance of events based on reward-related information is critical for animals to survive and to conduct social activities. The importance of the perirhinal cortex for reward-related information processing has been suggested. To examine whether or not neurons in this cortex represent reward information flexibly when a visual stimulus indicates either a rewarded or unrewarded outcome, neuronal activity in the macaque perirhinal cortex was examined using a conditional-association cued-reward task. The task design allowed us to study how the neuronal responses depended on the animal's prediction of whether it would or would not be rewarded. Two visual stimuli, a color stimulus as Cue1 followed by a pattern stimulus as Cue2, were sequentially presented. Each pattern stimulus was conditionally associated with both rewarded and unrewarded outcomes depending on the preceding color stimulus. We found an activity depending upon the two reward conditions during Cue2, i.e., pattern stimulus presentation. The response appeared after the response dependent upon the image identity of Cue2. The response delineating a specific cue sequence also appeared between the responses dependent upon the identity of Cue2 and reward conditions. Thus, when Cue1 sets the context for whether or not Cue2 indicates a reward, this region represents the meaning of Cue2, i.e., the reward conditions, independent of the identity of Cue2. These results suggest that neurons in the perirhinal cortex do more than associate a single stimulus with a reward to achieve flexible representations of reward information.

Differential encoding of factors influencing predicted reward value in monkey rostral anterior cingulate cortex.

BACKGROUND: The value of a predicted reward can be estimated based on the conjunction of both the intrinsic reward value and the length of time to obtain it. The question we addressed is how the two aspects, reward size and proximity to reward, influence the responses of neurons in rostral anterior cingulate cortex (rACC), a brain region thought to play an important role in reward processing. METHODS AND FINDINGS: We recorded from single neurons while two monkeys performed a multi-trial reward schedule task. The monkeys performed 1-4 sequential color discrimination trials to obtain a reward of 1-3 liquid drops. There were two task conditions, a valid cue condition, where the number of trials and reward amount were associated with visual cues, and a random cue condition, where the cue was picked from the cue set at random. In the valid cue condition, the neuronal firing is strongly modulated by the predicted reward proximity during the trials. Information about the predicted reward amount is almost absent at those times. In substantial subpopulations, the neuronal responses decreased or increased gradually through schedule progress to the predicted outcome. These two gradually modulating signals could be used to calculate the effect of time on the perception of reward value. In the random cue condition, little information about the reward proximity or reward amount is encoded during the course of the trial before reward delivery, but when the reward is actually delivered the responses reflect both the reward proximity and reward amount. CONCLUSIONS: Our results suggest that the rACC neurons encode information about reward proximity and amount in a manner that is dependent on utility of reward information. The manner in which the information is represented could be used in the moment-to-moment calculation of the effect of time and amount on predicted outcome value.

Role of temporal processing stages by inferior temporal neurons in facial recognition.

In this review, we focus on the role of temporal stages of encoded facial information in the visual system, which might enable the efficient determination of species, identity, and expression. Facial recognition is an important function of our brain and is known to be processed in the ventral visual pathway, where visual signals are processed through areas V1, V2, V4, and the inferior temporal (IT) cortex. In the IT cortex, neurons show selective responses to complex visual images such as faces, and at each stage along the pathway the stimulus selectivity of the neural responses becomes sharper, particularly in the later portion of the responses. In the IT cortex of the monkey, facial information is represented by different temporal stages of neural responses, as shown in our previous study: the initial transient response of face-responsive neurons represents information about global categories, i.e., human vs. monkey vs. simple shapes, whilst the later portion of these responses represents information about detailed facial categories, i.e., expression and/or identity. This suggests that the temporal stages of the neuronal firing pattern play an important role in the coding of visual stimuli, including faces. This type of coding may be a plausible mechanism underlying the temporal dynamics of recognition, including the process of detection/categorization followed by the identification of objects. Recent single-unit studies in monkeys have also provided evidence consistent with the important role of the temporal stages of encoded facial information. For example, view-invariant facial identity information is represented in the response at a later period within a region of face-selective neurons. Consistent with these findings, temporally modulated neural activity has also been observed in human studies. These results suggest a close correlation between the temporal processing stages of facial information by IT neurons and the temporal dynamics of face recognition.

Visualization of multi-neuron activity by simultaneous optimization of clustering and dimension reduction.

The recent development of arrays of microelectrodes have enabled simultaneous recordings of the activities of more than 100 neurons. However, it is difficult to visualize activity patterns across many neurons and gain some intuition about issues such as whether the patterns are related to some functions, e.g. perceptual categories. To explore the issues, we used a variational Bayes algorithm to perform clustering and dimension reduction simultaneously. We employed both artificial data and real neuron data to examine the performance of our algorithm. We obtained better clustering results than in a subspace that were obtained by principal component analysis.

Short-term memory trace in rapidly adapting synapses of inferior temporal cortex.

Visual short-term memory tasks depend upon both the inferior temporal cortex (ITC) and the prefrontal cortex (PFC). Activity in some neurons persists after the first (sample) stimulus is shown. This delay-period activity has been proposed as an important mechanism for working memory. In ITC neurons, intervening (nonmatching) stimuli wipe out the delay-period activity; hence, the role of ITC in memory must depend upon a different mechanism. Here, we look for a possible mechanism by contrasting memory effects in two architectonically different parts of ITC: area TE and the perirhinal cortex. We found that a large proportion (80%) of stimulus-selective neurons in area TE of macaque ITCs exhibit a memory effect during the stimulus interval. During a sequential delayed matching-to-sample task (DMS), the noise in the neuronal response to the test image was correlated with the noise in the neuronal response to the sample image. Neurons in perirhinal cortex did not show this correlation. These results led us to hypothesize that area TE contributes to short-term memory by acting as a matched filter. When the sample image appears, each TE neuron captures a static copy of its inputs by rapidly adjusting its synaptic weights to match the strength of their individual inputs. Input signals from subsequent images are multiplied by those synaptic weights, thereby computing a measure of the correlation between the past and present inputs. The total activity in area TE is sufficient to quantify the similarity between the two images. This matched filter theory provides an explanation of what is remembered, where the trace is stored, and how comparison is done across time, all without requiring delay period activity. Simulations of a matched filter model match the experimental results, suggesting that area TE neurons store a synaptic memory trace during short-term visual memory.

Cue and reward signals carried by monkey entorhinal cortex neurons during reward schedules.

Ablation of entorhinal/perirhinal cortices prevents learning associations between visual stimuli used as cues in reward schedules and the schedule state. Single neurons in perirhinal cortex are sensitive to associations between the cues and the reward schedules. To investigate whether neurons in the entorhinal cortex have similar sensitivities, we recorded single neuronal activity from two rhesus monkeys while the monkeys performed a visually cued reward schedule task. When the cue was related to the reward schedules, the monkeys made progressively fewer errors as the schedule state became closer to the reward state, showing that the monkeys were sensitive to the cue and the schedule state. Of 75 neurons recorded in the entorhinal cortex during task performance, about 30% responded. About half of these responded after cue presentation. When the relation of the cue to the reward schedules was random, the cue-related responses disappeared or lost their selectivity for schedule states. The responses of the entorhinal cortex neurons are similar to responses of perirhinal cortex neurons in that they are selective for the associative relationships between cues and reward schedules. However, they are particularly selective for the first trial of a new schedule, in contrast to perirhinal cortex where responsivity to all schedule states is seen. A different subpopulation of entorhinal neurons responded to the reward, unlike perirhinal neurons which respond solely to the cue. These results indicate that the entorhinal signals carry associative relationships between the visual cues and reward schedules, and between rewards and reward schedules that are not simply derived from perirhinal cortex by feed-forward serial processing.

Neuronal signals in the monkey basolateral amygdala during reward schedules.

The amygdala is critical for connecting emotional reactions with environmental events. We recorded neurons from the basolateral complex of two monkeys while they performed visually cued schedules of sequential color discrimination trials, with both valid and random cues. When the cues were valid, the visual cue, which was present throughout each trial, indicated how many trials remained to be successfully completed before a reward. Seventy-six percent of recorded neurons showed response selectivity, with the selectivity depending on some aspects of the current schedule. After a reward, when the monkeys knew that the upcoming cue would be valid, 88 of 246 (36%) neurons responded between schedules, seemingly anticipating the receiving information about the upcoming schedule length. When the cue appeared, 102 of 246 (41%) neurons became selective, at this point encoding information about whether the current trial was the only trial required or how many more trials are needed to obtain a reward. These cue-related responses had a median latency of 120 ms (just between the latencies in inferior temporal visual area TE and perirhinal cortex). When the monkey was releasing a touch bar to complete the trial correctly, 71 of 246 (29%) neurons responded, with responses in the rewarded trials being similar no matter which schedule was ending, thus being sensitive to the reward contingency. Finally, 39 of 246 (16%) neurons responded around the reward. We suggest that basolateral amygdala, by anticipating and then delineating the schedule and representing reward contingency, provide contextual information that is important for adjusting motivational level as a function of immediate behavior goals.

Categorical signals in a single-trial neuron activity of the inferotemporal cortex.

We developed an algorithm that decodes categorical signals from the single-trial activity of a neuronal population in the monkey inferotemporal cortex. We defined a global category (i.e. human faces vs. monkey faces vs. shape) and fine categories (i.e. human identity, monkey expression, and shape form) from the single-trial activity. The accuracy of estimation for the trials was roughly 100% for the global category and 88.1% for the fine categories. The accuracy of stimulus identification for the trials was 70.4%. These results suggest that signals concerning global and fine categories as well as object identification can be decoded using the single-trial activity of a neuronal population in the inferotemporal cortex.

Neuronal mechanisms encoding global-to-fine information in inferior-temporal cortex.

Sugase et al. found that global information is represented at the initial transient firing of a single face-responsive neuron in inferior-temporal (IT) cortex, and that finer information is represented at the subsequent sustained firing. A feed-forward model and an attractor network are conceivable models to reproduce this dynamics. The attractor network, specifically an associative memory model, is employed to elucidate the neuronal mechanisms producing the dynamics. The results obtained by computer simulations show that a state of neuronal population initially approaches to a mean state of similar memory patterns, and that it finally converges to a memory pattern. This dynamics qualitatively coincides with that of face-responsive neurons. The dynamics of a single neuron in the model also coincides with that of a single face-responsive neuron. Furthermore, we propose two physiological experiments and predict the results from our model. Both predicted results are not explainable by the feed-forward model. Therefore, if the results obtained by actual physiological experiments coincide with our predicted results, the attractor network might be the neuronal mechanisms producing the dynamics of face-responsive neurons.

Population dynamics of face-responsive neurons in the inferior temporal cortex.

Neurons in the inferior temporal (IT) cortex of monkeys respond selectively to complex visual stimuli, such as faces. Single neurons in the IT cortex encode different kinds of information about visual stimuli in their temporal firing patterns. To understand the temporal aspects of the information encoded at a population level in the IT cortex, we applied principal component analysis (PCA) to the responses of a population of neurons. The responses of each neuron were recorded while visual stimuli that consisted of geometric shapes and faces of humans and monkeys were presented. We found that global categorization, i.e. human faces versus monkey faces versus shapes, occurred in the earlier part of the population response, and that fine categorization occurred within each member of the global category in the later part of the population response. A cluster analysis, a mixture of Gaussians analysis, confirmed that the clusters in the earlier part of the responses represented the global category. Moreover, the clusters in the earlier part separated into sub-clusters corresponding to either human identity or monkey expression in the later part of the responses, and the global categorization was maintained even after the appearance of the sub-clusters. The results suggest that a hierarchical relationship of the test stimuli is represented temporally by the population response of IT neurons.